<html>
<head><meta charset="utf-8"><title>2020-02-13 meeting · t-compiler/wg-parallel-rustc · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/index.html">t-compiler/wg-parallel-rustc</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html">2020-02-13 meeting</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="188103235"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188103235" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188103235">(Feb 13 2020 at 12:56)</a>:</h4>
<p>I am likely not going to be able to make it today, but am unopposed if y'all want to meet.</p>



<a name="188120432"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188120432" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188120432">(Feb 13 2020 at 16:03)</a>:</h4>
<p>I can't really make this time that much <em>but</em></p>



<a name="188120612"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188120612" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188120612">(Feb 13 2020 at 16:05)</a>:</h4>
<p>I definitely feel we need to agree on a plan and immediate steps</p>



<a name="188120629"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188120629" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188120629">(Feb 13 2020 at 16:05)</a>:</h4>
<p>It still feels to me that we're moving in a few directions</p>



<a name="188120810"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188120810" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188120810">(Feb 13 2020 at 16:07)</a>:</h4>
<p>As a related aside, one thing that <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> and I were talking about was the idea of doing a "compile team lecture series" video (haven't done one of those in a while) about how the current support is working at a high level. <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> felt it would be useful to them. I felt like I could do some of that, but I don't think I'd be the best choice, certainly not for all of it, but maybe a <span class="user-mention" data-user-id="116122">@simulacrum</span> / <span class="user-mention" data-user-id="116015">@Alex Crichton</span> / <span class="user-mention" data-user-id="116466">@Zoxc</span> there could sort of be a joint effort to describe state?</p>



<a name="188120826"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188120826" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188120826">(Feb 13 2020 at 16:07)</a>:</h4>
<p>i.e., somebody who runs but maybe tag teams <span class="user-mention" data-user-id="116015">@Alex Crichton</span> to explain job server or something :P</p>



<a name="188121199"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188121199" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188121199">(Feb 13 2020 at 16:11)</a>:</h4>
<p>But the other big issue I think is more about the overall philosophy and design we are shooting for. I raised this a bit in <a class="stream-topic" data-stream-id="187679" href="/#narrow/stream/187679-t-compiler.2Fwg-parallel-rustc/topic/rustc-rayon.20extension">#t-compiler/wg-parallel-rustc &gt; rustc-rayon extension</a>, for example -- I think that in all the "compiler team" meetings I've been at, we've had a philosophy that we want to move to "vanilla rayon", and generally shoot for a simple setup overall to start, but I'm not sure <span class="user-mention" data-user-id="116466">@Zoxc</span> what your take is on that. My feeling is you don't agree and it seems like we should hammer this out. =) I can imagine for example that maybe a simpler setup is just not going to get the perf we need, and I think e.g. the discussion about panic handling sounded relevant. I guess most of all I'd like to get some picture of the high-level setup we are shooting for in order to ship, so that we can place PRs in that context.</p>



<a name="188121260"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188121260" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188121260">(Feb 13 2020 at 16:11)</a>:</h4>
<p>I owe Santiago a doodle poll for that as well as a bisect pr. I also have found that this time is not great for me.</p>
<p>I guess we can try to find a time to do such a session - to be honest, I can explain jobserver myself I think. Everything else I don't have a good handle on myself.</p>



<a name="188121313"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188121313" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188121313">(Feb 13 2020 at 16:12)</a>:</h4>
<p>Er, bisect rustc explainer</p>



<a name="188121693"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188121693" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188121693">(Feb 13 2020 at 16:15)</a>:</h4>
<p>Yes, I agree some consensus and hammering on overall design is necessary.</p>



<a name="188122615"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188122615" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188122615">(Feb 13 2020 at 16:23)</a>:</h4>
<p>yeah, I've mentioned the idea to Mark too</p>



<a name="188122696"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188122696" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188122696">(Feb 13 2020 at 16:24)</a>:</h4>
<p>also <span class="user-mention" data-user-id="116122">@simulacrum</span> if you don't have a lot of time forget about the bisect thing, I can just sit down and prepare something and explain the rest of the people that need that</p>



<a name="188122700"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188122700" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188122700">(Feb 13 2020 at 16:24)</a>:</h4>
<p>don't worry</p>



<a name="188122724"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188122724" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188122724">(Feb 13 2020 at 16:24)</a>:</h4>
<p>let me open a separate thread for that</p>



<a name="188124103"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188124103" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188124103">(Feb 13 2020 at 16:37)</a>:</h4>
<p>I'm on the subway but I was thinking about this and I wanted to say -- I have the feeling <span class="user-mention" data-user-id="116466">@Zoxc</span> that you do have something of  a plan in your mind, or at least an idea of constraints that we're not all fully aware of. Maybe a good place to start would be that we can talk about the path that <em>you</em> have in mind (without trying to forge a consensus or pick anything yet), just so that we all understand each other more deeply?</p>



<a name="188128231"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188128231" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188128231">(Feb 13 2020 at 17:18)</a>:</h4>
<p>That does seem a bit abstract =P</p>



<a name="188129661"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188129661" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188129661">(Feb 13 2020 at 17:34)</a>:</h4>
<p>oof sorry on west coast where this time is a bit early for me, but catching up on stuff now. Always happy to chat about jobserver though! (and agreed on sync'ing with directions)</p>



<a name="188223419"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188223419" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188223419">(Feb 14 2020 at 16:45)</a>:</h4>
<p>At some point, <span class="user-mention" data-user-id="116122">@simulacrum</span> and I were tinkering with a <a href="https://hackmd.io/BxKmG7ItQ2K8SLUdmu1kNQ" target="_blank" title="https://hackmd.io/BxKmG7ItQ2K8SLUdmu1kNQ">hackmd</a> trying to write out a proposal with core goals and some questions/answers</p>



<a name="188223642"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188223642" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188223642">(Feb 14 2020 at 16:48)</a>:</h4>
<p><span class="user-mention" data-user-id="116466">@Zoxc</span> to make it concrete, maybe we can just try to lob specific questions your way</p>



<a name="188223645"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188223645" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188223645">(Feb 14 2020 at 16:48)</a>:</h4>
<p>to start I would ask</p>



<a name="188223711"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188223711" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188223711">(Feb 14 2020 at 16:49)</a>:</h4>
<ul>
<li>What do you see as the next steps to improve parallel performance, and/or what blockers are on your mind?</li>
<li>What criteria was used to decide where we insert parallelism like <code>par_iter</code> or <code>parallel!</code>?</li>
</ul>



<a name="188231803"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188231803" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188231803">(Feb 14 2020 at 18:24)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116009">nikomatsakis</span> <a href="#narrow/stream/187679-t-compiler.2Fwg-parallel-rustc/topic/2020-02-13.20meeting/near/188223711" title="#narrow/stream/187679-t-compiler.2Fwg-parallel-rustc/topic/2020-02-13.20meeting/near/188223711">said</a>:</p>
<blockquote>
<ul>
<li>What do you see as the next steps to improve parallel performance, and/or what blockers are on your mind?</li>
</ul>
</blockquote>
<ul>
<li>move the query system earlier in the compiler (this aligns well with incremental compilation)</li>
<li>use a hashmap with lock-free lookup as the query cache</li>
<li>the ability to spawn a closure to run in the background which can capture <code>TyCtxt</code> would be very useful (to encode incremental state in the background and prefetching queries (like parsing a file when we see <code>mod foo;</code>))</li>
<li>make miri use less global state (I haven't looked at it in a while, but I suspect no improvements)</li>
</ul>
<p>In terms of avoiding regressions we might need to a flag that turns on parallelism/atomics (possibly on demand). We could turn off parallelism when building dependencies for example, which is more likely to saturate all cores.</p>



<a name="188232050"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188232050" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188232050">(Feb 14 2020 at 18:27)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116009">nikomatsakis</span> <a href="#narrow/stream/187679-t-compiler.2Fwg-parallel-rustc/topic/2020-02-13.20meeting/near/188223711" title="#narrow/stream/187679-t-compiler.2Fwg-parallel-rustc/topic/2020-02-13.20meeting/near/188223711">said</a>:</p>
<blockquote>
<ul>
<li>What criteria was used to decide where we insert parallelism like <code>par_iter</code> or <code>parallel!</code>?</li>
</ul>
</blockquote>
<p>Inserting it must improve performance with multiple threads (so there must be free cores at that point), should not affect single thread performance much, and it also must be efficient so CPU cores are not wasted on a single <code>rustc</code> process when it could be doing something else.</p>



<a name="188237898"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188237898" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188237898">(Feb 14 2020 at 19:27)</a>:</h4>
<p>How were locations for par_iter/parallel! as they exist today found? Those seem like good criteria for why not to insert it in some sense (or what to look at in detail when doing so), but how were these specific spots found? Just looking through the whole codebase?</p>



<a name="188238828"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188238828" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoxc <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188238828">(Feb 14 2020 at 19:37)</a>:</h4>
<p>Guided by <code>-Z time-passes</code> mostly, and my later PRs by <code>-Z self-profile</code>.</p>



<a name="188238935"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188238935" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188238935">(Feb 14 2020 at 19:38)</a>:</h4>
<p>So there is no guiding principle beyond simple performance, if I follow you? i.e., we aren't trying to parallelize because of X where X is not performance?</p>



<a name="188238975"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13%20meeting/near/188238975" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/187679-t-compiler/wg-parallel-rustc/topic/2020-02-13.20meeting.html#188238975">(Feb 14 2020 at 19:38)</a>:</h4>
<p>(I'm struggling to come up with an example, but e.g., I could imagine some sort of reason like 'this is a loop and it must be parallel, even if we have no data to suggest so')</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>